Getbhavcopy: Free NSE and BSE data downloader

Getbhavcopy version 2.1.3a released

Fri, 06 Jan 2012 09:10:00 +0000

This new version fixes the following issues

Startup issues that were happening since past few days
501 not implemented error for VIX data download
501 not implemented error for Open Interest data downloadfor NSE Equities.

Enjoy the new version! If you feel Getbhavcopy is worth it, please donate.

I Need your Help.

Do you

Regularly use Getbhavcopy to download either NSE or BSE data and
Use Metastock as your Technical analysis tool (i.e. import Getbhavcopy data to Metastock) and
Keep your Metastock database up to date with Splits and Bonus declerations as well as company symbol/name changes (this is most important)

Only if you satisfy all three conditions, I need your help.

Can you zip your Metastock database directory (seperate zip files for NSE and BSE EOD data), and then upload to 4shared.com and share the link in the comments section of this post?

Your help will be much appreciated. My historical data is up to 2008 and since I am in USA for past few years and bit busy have not kept pace with updating my data.

Blessed with a baby girl

Sun, 24 Apr 2011 00:00:00 +0000

For the past few days I have been a lot busy, so taking the first opportunity to share the most important joy of my life with the Getbhavcopy community members, who have now become an extended family.

My wife delivered a baby girl on April 7th. Both mother and baby are fine.

We have named her Vidhi. Sharing her picture.

Getbhavcopy version 2.1.2a released

Wed, 16 Feb 2011 08:35:00 +0000

NSE has updated the NSE indices data page with some additional checks that blocked Getbhavcopy. It also changed the HTML code a bit causing the Getbhavcopy parser to fail.

This has now been fixed in version 2.1.2a.

I understand the frustration you all feel when Getbhavcopy does not work, that itself speaks for the value of this software, but that I would like to make a few points clear to the community due the recent allegations I have come across

The delay in fixing this issue is no way related to the decreased donation flow recently. As a result i have not asked for any donation this time. I have renewed the server 2 weeks back.
I would expect you to understand that I have a family and a personal life too. I have financial responsibilites to feed them as a result my first priority is my job and not Getbhavcopy.
I have a family and they are humans, and just like any other human being they can also have health conditions that warrent my attention. My family comes before Getbhavcopy.
Getbhavcopy is my hobby, and I am sharing it with you so that you all can benefit from it. In such a scenario it is not an obligation on my part to fix this as soon as an issue arises or to release any part of my code to any one. These decisions are solely my discretion.

Request the community members to please be more considerate about the points I made above in case there are future disruptions.

Please do report the issues (in large numbers) however to warrent my attention.

NSE indices issues ?

Sat, 12 Feb 2011 04:35:00 +0000

First and foremost, apologies for not responding this long. I am really busy with my office work and i am not getting any time to work on Getbhavcopy. Moreover there are few things on the personal front that are occupying my other time.

I understand the plight of all the Getbhavcopy users but I hope you will understand my current situation.

I will try to look into this over the weekend if possible, but please do not take this as a commitment.

P.S. There is no connection between not receiving donations lately and my inability to fix this issue. It is purely due to sever lack of time and sleep.

Getbhavcopy version 2.1.1a released

Sat, 06 Nov 2010 09:58:00 +0000

First of all apologies for the inconvenience caused due to the 403 Forbidden error message that started showing up recently. NSE has made some changes on their server side that blocked Getbhavcopy (and other similar downloaders) from downloading data.

This issue has been fixed in this alpha version and Getbhavcopy is able to download as before from NSE and BSE. I will soon some with a proper version after rearrenging my code and some additional features.

I would also like to wish all users of Getbhavcopy a very happy Diwali.Hope you like this small Diwali gift from my end.

Regards,

Hemen

P.S. I would like to remind users that my server renewal is approachin and is due Jan 2011. I would appreciate if you can spare some donation in return for my services.

403 Forbidden Error ? Are you facing this ?

Thu, 04 Nov 2010 02:37:00 +0000

Looks like NSE has got smart and has enabled some additional checks that is blocking Getbhavcopy from downloading the files.

Right now i dont know what needs to be done to fix this. I am working on this. I will keep updating this blog post to keep everyone appraised about the progress.

Thanks for bearing with me.

Updates also on my Twitter page. Follow me to keep track of the progress.

Update 1: 4 Nov 2010, 1:38 AM PST : Hemen

I tried spending about 4 hours trying to understand how the NSE server is able to differentiate between the requests sent by Browser from those sent by Getbhavcopy. As of now I have not been able to resolve it. Will continue investigating tomorrow.

Update 2: 4 Nov 2010, 11:59 AM PST : Hemen

Testman suggested the following fix.

Hi Hemen,Add following to your http request headers:Referer: http://www.nseindia.com/archives/archives.htmUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 (.NET CLR 3.5.30729)Give 15 seconds gap between two requests.Hope it helps

I will try to code this tonight and report if this fixes the issue.

Update 3: 6 Nov 2010, 02:05 AM PST : Hemen

Good news for all you getbhavcopy fans. The 403 Forbidden error has been fixed. Here is the proof

It is almost 2 in the morning and tomorrow need to get up early for Diwali New Year. So will be going off to bed.

I will release 2.1.1a (Alpha release) tomorrow as I need to clean up some code.

Update 4 : 6 Nov 2010 03:04 AM PST: Hemen

Could not wait till tomorrow to release it. So releasing it now itself. 2.1.1a version is availabe for you on the Downloads Page.

It fixes the 403 Forbidden error and Getbhavcopy is working again !!

Wish you all a happy Diwali.

Getbhavcopy download issues - Part 1

Sat, 23 Oct 2010 03:17:00 +0000

First of all I would like to apologise all the members of Getbhavcopy community for the delays caused in addressing your issues. I have been really very busy these days with my office work that I am unable to dedicate any time to address your woes.

Saying that, I am really impressed with the way community members are coming together to help each other out. I never expected Getbhavcopy to become so popular.

Mr. MurthyMSR (who has been a long time supporter) has made a very good post consolidating the issues faced by users and the some ideas how he got it to work. I am converting that post into a blog for all users.

The download problems refrerred to occur quite often, and can be minimized if not totally avaoided by following just ONE simple rule.

The errors reported could be because of one or more reasons like the ones given below. I too faced these problems as reflected in the various logs posted. Now-a-days, such occurances are very rare for me. But once they occur, the solution lies in just trying again after some time.

Possible reasons could also be one or more of the following.

Your net connection may be slow or not connected.
When you press the download button, the program checks if your installed version is the latest by checking with the GetBhavCopy site. If your version is old, you will be prompted to update to the latest version and download will not be attempted. But if the GBC site is down, this can produce an error.
One of the varous links by which you are connected to the resource may be down.
NSE & BSE sites may be down or the destination uRL's may not be updated or too many people arew requesting for that information beyond the capacity limits set within the servers.
And may be some more, I am not able to visualize.

All these reasons are external to the GBC program. Of course GBC has some issues to be sorted out but none of these.

Please note:

GBC fetches data from NOT ONE but, MANY sources (addresses) within NSE & BSE web sites.
All these souces are updated at the same time.
The first source (Bhav Copy) may be ready by 6:30 PM but all other sources take a few hours more.
After observong this phenomenon of failure for many weeks/months. Now-a-days, I download only after my dinner before going to bed, i.e., after 10 PM.
I consider a download successful when ALL repeat ALL the data is downloaded completely. This includes:

NSE stock Bhav copy, stock open interest, each of the NSE indices, their volumes,
NSE futures bhav copy
BSE stocks, each of the BSE indices

I checked the completeness by veryfying data in each of the above categories manually initially and later by AB.
This completeness of data can be checked with AB as below after importing the downloaded data into AB.

Run AB >> Tools >> Database purity ...
Select a very actively traded NSE stock like Reliance.as reference symbol.
Deselect all check boxes on the left side EXCEPT 'Report missing quotes'
On the right side, within the 'Apply to' box, select "All symbols" and quotes Last n= , by this you are checking if any quotes are missing in todays bhav copies of NSE & BSE by comparing it with the NSE stock 'RELIANCE'
If the list of missing quotes you find only some very thinly traded stocks and do NOT find any Index, you may consider the download complete.
Repeat this excesize this time by taking an actively traded BSE stock like reliance by it's numerical code.
Again if the list of missing quotes you find only some very thinly traded stocks and do NOT find any Index, you may confirm that the download complete.
I have done this excercize many times and found that the download is complete.

You also may do this excercize to get that confidence about the data completion.

FOR AN ERROR FREE AND COMPLETE DOWNLOAD, DO IT AFTER 10 PM . 90% CHANCE THAT THE DOWNLOAD IS ERROR FREE. OTHERWISE, REPEAT AFTER 5 MINUTES.

YOU WILL BE THROUGH WITH THE DOWNLOAD ERROR FREE AND COMPLETE.

( sorry for caps)

If you are further keen, do the AB database purity check as outlined above to get confidence in what all I said. If you still get errors, then report at this forum for help by the developer.

Hope my observations will be helpful at least some people.

I am also a user like you, not connected with GBC program in any way. I too faced many problems & irritations like the ones mentioned by you and found that this is the way to get the best of GBC.

I request hemen to go through this post offer his comments on the contents. I feel that the error log may contain messages in plain English that point out the problem in a way a normal investor/trader ( not familiar with the technical jargaon like '404 not found,.) can easily understand.

I am using GBC without errors and hope that everybody too gets it that way.

Thanks to hemen for this program.

murthymsr

Regards.

Thanks a lot sir.

I will provide my consolidated analysis in the Part 2 of this post over this weekend.

Once again sorry for the inconvenience and thanks for sticking by.

Getbhavcopy version 2.1.0 released

Mon, 28 Jun 2010 04:23:00 +0000

It has been some time since I released version 2.1.0a, and no major issues have been reported. So I am making it as a released candidate and will be discontinuinig the 2.0.7 and 2.1.0.a versions.

To understand what new features have been implemented in the version 2.1.0, in case you are upgrading from version 2.0.7, please read the release post for version 2.1.0a

As compared to version 2.1.0a I have fixed the DPI resolution issue for the options form, where it does not appear properly for those systems runnung on 120 DPI setting. Since most users work on 96 DPI you may not notice it.

Anyway, I urge all users to update to version 2.1.0. The download links are availabe on the Download Page.

Once again I urge users to take benefit of the new features added in this version. Also the donations have considerably dried up recently. Urge people to please donate.

Moved back to USA

Wed, 21 Apr 2010 04:55:00 +0000

For those who are regulars at my site and are wondering why I have noot been posting or answering to any queries lately, i would like to inform that it has been really hectic for me in the last two to three months and I have not had the time to respond to visit this site.

I apologise if any inconvenience caused to you.

Also I was busy with my move back to USA for a work related project, and was busy setting up my home internet so that once again I can start working on Getbhavcopy.

Since a major part of development of Getbhavcopy happened in my last visit to USA. I hope to continue the same and keep adding new features.

Hope I get time to visit the site more often and be more responsive. Till then keep using Getbhavcopy.

Getbhavcopy version 2.1.0a released

Tue, 16 Feb 2010 14:52:00 +0000

Update: Version 2.1.0 has been released. Please refer to the release post

There is not much activity on the forums these days. May be everyone is really busy like I have been over the past few weeks (causing me to delay this release which i promised to do on 31 Jan). Anyway, i hope this new release spices things up a bit.

First of all I would like to draw your attention to the 'a' after the version number for this relese. If you noticed, none of the earlier releases had that. The 'a' stands for Alpha Release, which means that this is not a stable release (also called as a Release Candidate). In other words this release is not tested fully.

There are considerable new features that have been implemented, but I cannot test all of them in different environments that Getbhavcopy may be subjected to run. Hence there could be some potential bugs or issues that you can notice. This is more of a test version (or an early peek) to get your feedback to improve Getbhavcopy.

What's new ?

NSE futures data download.

Most of you have been asking for this for a long time, so here is the much needed feature for you. Both FUTIDX and FUTSTX data is downloaded for near, next and far series. Userdefined symbols can be added to these series. By default -I, -II and -III are appeneded to the symbols respectively. The cycles will also rollover to the previous ones providing a continuous chart for analysis.

Support for Proxy authentication to avoid 407 errors.

This is totally untested and I really need help of those who are behind proxy to test this. Plese post your findings in the forums.

Multithreading support.

Have you observed Getbhavcopy UI kind of hanging when it is downloading data? Well not any more as now the UI update and data download threads are different.

NSE Indev volume calculate option.

NSE Indices data has now started providing volume information too. This was not available earliler, hence Getbhavcopy used to calculate it. Now you have the option to avoid this calculation which results in less data to be downloaded (so faster download). However NSE may turn it off anytime (as seen in past) so you still have the option to enable calculation to go back the old way.

VIX data download.

NSE VIX data will also be downloaded.

New Options form

The Options form has been removed from the main download window and converted into a wizard kind of the form. You need to click the Next button from all tabs till you reach the final tab that provides the Save option.

Download

The download link is on the Getbhavcopy Download page. Check the Alpha Release section.

Install Instructions

To install download the zip file and read the Readme.txt file in the zip for more details. It contains very important points that should be read first before using the alhpa version (even for the seasoned Getbhavcopy users)

Donation

Finally, the donations have really dried up. Haven't recieved anything since past two months. So guys please tend to donate something, it really helps keeping me motivated.

Finally, test this version and report your issues on Forums. Also mention the Getbhavcopy version henceforth when posting in forums as there are two active versions now.

The current stable version 2.0.7 will continue to work till this becomes stable