Getbhavcopy: Free NSE and BSE data downloader

Blog

« Back to Blog « Older Entry | Newer Entry »

403 Forbidden Error ? Are you facing this ?

Posted on November 3, 2010 at 10:37 PM

Looks like NSE has got smart and has enabled some additional checks that is blocking Getbhavcopy from downloading the files.


Right now i dont know what needs to be done to fix this. I am working on this. I will keep updating this blog post to keep everyone appraised about the progress.


Thanks for bearing with me.


Updates also on my https://twitter.com/getbhavcopy" target="_blank">Twitter page. Follow me to keep track of the progress.


Update 1: 4 Nov 2010, 1:38 AM PST : Hemen


I tried spending about 4 hours trying to understand how the NSE server is able to differentiate between the requests sent by Browser from those sent by Getbhavcopy. As of now I have not been able to resolve it. Will continue investigating tomorrow.


Update 2: 4 Nov 2010, 11:59 AM PST : Hemen


Testman suggested the following fix.

Hi Hemen,Add following to your http request headers:Referer: http://www.nseindia.com/archives/archives.htmUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 (.NET CLR 3.5.30729)Give 15 seconds gap between two requests.Hope it helps


I will try to code this tonight and report if this fixes the issue.


Update 3: 6 Nov 2010, 02:05 AM PST : Hemen


Good news for all you getbhavcopy fans. The 403 Forbidden error has been fixed. Here is the proof






It is almost 2 in the morning and tomorrow need to get up early for Diwali New Year. So will be going off to bed.


I will release 2.1.1a (Alpha release) tomorrow as I need to clean up some code.


Update 4 : 6 Nov 2010 03:04 AM PST: Hemen


Could not wait till tomorrow to release it. So releasing it now itself. 2.1.1a version is availabe for you on the Downloads Page.


It fixes the 403 Forbidden error and Getbhavcopy is working again !!


Wish you all a happy Diwali.

Categories: None

Post a Comment

Oops!

Oops, you forgot something.

Oops!

The words you entered did not match the given text. Please try again.

You must be a member to comment on this page. Sign In or Register

28 Comments

Reply falgunpachal
12:33 PM on February 15, 2011 
HI FRIENDS , I M NOT ABLE TO DOWN LOAD NSE DATA ,EVERY TIME THERE IS ERROR
KINDLY GIVE SOLUTION
Message ********* Downloading Data for 15-02-2011 *********
Message Downloading NSE Equity Bhavcopy
Message Downloading index data for NSENIFTY
Error Error sending Index Data HTTP POST for NSENIFTY
Error The remote server returned an error: (500) Internal Server Error.
Message Downloading index data for NIFTYJUNIOR
Error Error sending Index Data HTTP POST for NIFTYJUNIOR
Error The remote server returned an error: (500) Internal Server Error.
Message Downloading index data for BANKNIFTY
Error Error sending Index Data HTTP POST for BANKNIFTY
Error The remote server returned an error: (500) Internal Server Error.
Message Downloading index data for NSE100
Error Error sending Index Data HTTP POST for NSE100
Error The remote server returned an error: (500) Internal Server Error.
Message Downloading index data for NSEMIDCAP
Error Error sending Index Data HTTP POST for NSEMIDCAP
Error The remote server returned an error: (500) Internal Server Error.
Message Downloading index data for NSEIT
Error Error sending Index Data HTTP POST for NSEIT
Error The remote server returned an error: (500) Internal Server Error.
Message Downloading index data for NSE500
Error Error sending Index Data HTTP POST for NSE500
Error The remote server returned an error: (500) Internal Server Error.
Message Downloading index data for NSEDEFTY
Error Error sending Index Data HTTP POST for NSEDEFTY
Error The remote server returned an error: (500) Internal Server Error.
Message Downloading index data for MIDCAP50
Error Error sending Index Data HTTP POST for MIDCAP50
Error The remote server returned an error: (500) Internal Server Error.
Message Downloading index data for VIX
Message Processing downloaded data
Message ********* NSE Equity Bhavcopy file generated *********
Message Downloading NSE Futures Bhavcopy
Message Processing downloaded data
Message ********* NSE Futures Bhavcopy file generated *********
Message Downloading BSE Equity Bhavcopy
Message Downloading index data for SENSEX
Message Downloading index data for MIDCAP
Message Downloading index data for SMLCAP
Message Downloading index data for BSE-100
Message Downloading index data for BSE-200
Message Downloading index data for BSE-500
Message Downloading index data for AUTO
Message Downloading index data for BANKEX
Message Downloading index data for CONSDURBL
Message Downloading index data for CAPGOODS
Message Downloading index data for FMCG
Message Downloading index data for HLTHCARE
Message Downloading index data for IT
Message Downloading index data for METAL
Message Downloading index data for OILGAS
Message Downloading index data for POWER
Message Downloading index data for PSU
Message Downloading index data for REALTY
Message Downloading index data for TECK
Message Downloading index data for DOL-30
Message Downloading index data for DOL-100
Message Downloading index data for DOL-200
Message Processing downloaded data
Message ********* BSE Equity Bhavcopy file generated *********
Message ********* Data download complete *********
Reply Piyush Shah
06:05 AM on February 01, 2011 
Getting problem in downloading since last two days & also getting error in downloading Historical Data of BSE from 01-01-2008 to 30-06-2008

Message ********* Downloading Data for 28-01-2011 *********
Message Downloading NSE Equity Bhavcopy
Message Downloading index data for NSENIFTY
Error NSE server did not return index data for NSENIFTY
Message Downloading index data for NIFTYJUNIOR
Error NSE server did not return index data for NIFTYJUNIOR
Message Downloading index data for BANKNIFTY
Error NSE server did not return index data for BANKNIFTY
Message Downloading index data for NSE100
Error NSE server did not return index data for NSE100
Message Downloading index data for NSEMIDCAP
Error NSE server did not return index data for NSEMIDCAP
Message Downloading index data for NSEIT
Error NSE server did not return index data for NSEIT
Message Downloading index data for NSE500
Error NSE server did not return index data for NSE500
Message Downloading index data for NSEDEFTY
Error NSE server did not return index data for NSEDEFTY
Message Downloading index data for MIDCAP50
Error NSE server did not return index data for MIDCAP50
Message Downloading index data for VIX
Message Processing downloaded data
Message ********* NSE Equity Bhavcopy file generated *********
Message ********* Downloading Data for 31-01-2011 *********
Message Downloading NSE Equity Bhavcopy
Message Downloading index data for NSENIFTY
Error NSE server did not return index data for NSENIFTY
Message Downloading index data for NIFTYJUNIOR
Error NSE server did not return index data for NIFTYJUNIOR
Message Downloading index data for BANKNIFTY
Error NSE server did not return index data for BANKNIFTY
Message Downloading index data for NSE100
Error NSE server did not return index data for NSE100
Message Downloading index data for NSEMIDCAP
Error NSE server did not return index data for NSEMIDCAP
Message Downloading index data for NSEIT
Error NSE server did not return index data for NSEIT
Message Downloading index data for NSE500
Error NSE server did not return index data for NSE500
Message Downloading index data for NSEDEFTY
Error NSE server did not return index data for NSEDEFTY
Message Downloading index data for MIDCAP50
Error NSE server did not return index data for MIDCAP50
Message Downloading index data for VIX
Message Processing downloaded data
Message ********* NSE Equity Bhavcopy file generated *********
Message ********* Data download complete *********
Reply b.raju72
12:19 PM on November 10, 2010 
Thanks a lot.
Reply Raj
06:11 AM on November 08, 2010 
Thank you very much for your effort.
Reply kkseal
02:50 AM on November 07, 2010 
Thanks to Hemen's efforts it's truly a Happy Deewali now for all GBC users.
Cheers!!!!
Reply PS
11:37 PM on November 06, 2010 
Friends, sorry for the bother, seemed to be BSE server related issue, downloaded fine this morning,
Regards.
Reply PS
10:18 PM on November 06, 2010 
Can someone please post the BSE txt file for 5th Nov, 2010?
Reply adatta78@gmail.com
09:40 AM on November 06, 2010 
Respected Hemen Sir,
Many many thanks to you for the modification.
This Diwali really lighten up our life
Thank you again.

Regards
Reply PS
08:31 AM on November 06, 2010 
With the newer version 2.1.1a all other data from 1st to 5th Nov downloads fine except BSE for 5th Nov, 2010, here is the log

Message ********* Starting Getbhavcopy *********
Message Settings Saved
Message ********* Downloading Data for 05-11-2010 *********
Message Downloading BSE Equity Bhavcopy
Error Could not extract BSE zip file.
Error Cannot find central directory
Message Downloading index data for SENSEX
Message Downloading index data for MIDCAP
Message Downloading index data for SMLCAP
Message Downloading index data for BSE-100
Message Downloading index data for BSE-200
Message Downloading index data for BSE-500
Message Downloading index data for AUTO
Message Downloading index data for BANKEX
Message Downloading index data for CONSDURBL
Message Downloading index data for CAPGOODS
Message Downloading index data for FMCG
Message Downloading index data for HLTHCARE
Message Downloading index data for IT
Message Downloading index data for METAL
Message Downloading index data for OILGAS
Message Downloading index data for POWER
Message Downloading index data for PSU
Message Downloading index data for REALTY
Message Downloading index data for TECK
Message Downloading index data for DOL-30
Message Downloading index data for DOL-100
Message Downloading index data for DOL-200
Message Processing downloaded data
Error BSE Processor cannot open temporary bhavcopy file
Error Could not find file 'O:\GBC\getbhavcopy\temp\BSEbhav.gbc'.
Message ********* Data download complete *********

I have retried on different PCs, but same issue, please advise.
Regards.
Reply murthymsr
01:01 AM on November 06, 2010 
Dear Hemen,
If you like the idea, you may release a temporary version with just the offline mode to mitigate the suffering of GBC users on lines similar to my exercise, by bypassing all the NSEIndia browser requests.

In the meantime, you can work out a permanent & full solution.

Thanks & regards.
murthymsr
Reply murthymsr
11:47 PM on November 05, 2010 
YES, it is working !!!

Abstract of today's (Murat Trading) Bhav copy (first few lines) obtained with present GBC version as below:

20MICRONS,20101105,56.7,57,56.05,56.55,8330
3IINFOTECH,20101105,68.5,68.5,66.85,67.05,183101
3MINDIA,20101105,4048.95,4048.95,3903,3973.15,152
AARTIDRUGS,20101105,158.8,158.8,154,155.65,23944
AARTIIND,20101105,77,79,76.05,76.8,105329

Follow getting these results, please follow this procedure strictly in that order:

1) download from NSE bhav copy of any day of your interest, but only one at a time and save as NSEBhav.gbc on to your desktop.

1) start bhav copy.

2) In windows explorer view, observe the NSEBhav.gbc in the temp folder of GBC. This is the old one.

2) Press the download button. The from & to Dates are of no consequence. Observe that the old NSEBhav.gbc file being deleted in temp folder of GBC. use refresh to observe this deletion.

2) The program attempts to download data for each index one by one and fails. You will have many 403 errors. Takes a few minutes. Be patient.

3) During this time, copy the NSEBhav.gbc file from your desktop and paste in your temp folder of GBC. This is very important.

4) Afer some more minutes, you will get the last three lines of log window as:

Message Processing downloaded data
Message ********* NSE Equity Bhavcopy file generated *********
Message ********* Data download complete *********

AND The process is COMPLETED & you have the file for the specified name (2010-11-05-NSE-EQ.txt in your NSE folder.

You will find the converted bhav copy ready to be imported into your TA software.

But please note that ALL indice data is missing. Delivery volume for each stock (as Open Interest) is missing. But at least this much you are getting till the modifications are done by Hemen., Be happy.

Happy Diwali & New Year to Hemen & all users of GBC.

murthymsr
Reply Loganathan B
06:06 PM on November 05, 2010 
Hi Heman, while updating GBC pl add NSEREALTY and NSEINFRA index to be downloaded. Or you can keep a menu format for selecting index lists.
Reply murthymsr
01:29 PM on November 05, 2010 
BIJOY says...
data down loader of joy is somewhat able to download eod datas..volumediggers.com is the site.But we love getbhavcopy by hemen..we are used to it.Hemen when datadownloader is capable of downloading why getbhavcopy is forbidden...plzz sort out the problem.Bijoy


Hi bijoy,

Migration from one downloader to another is a very painful task and may be done only if an absolute must.

The way the symbols are given for a stock is at variance. This is particularly so for indices and may be futures also. BSE sectors is one that will be totally missed. At least for that reason, one still have to use GBC. So instead, one can wait for a few days more, hoping for a positive solution.

In the meantime, some members are posting EOD data converted and the same may be used. It is in the interest of all to post in this forum only the EOD converted data which is totally compatible with the data being provided by GBC. If it a way different, the format file used to import into AB may have to be changed.

Not sure if anybody is providing NSE stock EOD data with the OI column filled with delivery volume.

murthymsr
Reply Sunil
11:40 AM on November 05, 2010 
murthymsr says...
Dear Hemen,

I was trying to understand the reasons behind the changes made by NSE resulting in the failure of DD to function normally and the type of modifications required to be done to

Downloaders to bring them back on track.

Looks like that NSE does not want to allow data downloads through SW, but wants to allow through browser only. This may be because the load on their servers is increasing

particularly during market hours.

Code in all downloaders, one by one, will be modified to imitate the browser request and downloaders will start function as before and load on NSE servers will reach the

previous valus and NSE may start looking out for other ways to prevent sw downloads.

It is unfortunate that NSE could have augmented their server capacity instead of restricting sw downloads.

Now to help NSE to reduce the load on their servers during market hours and to serve as a back-up plan, I wish;

1) More than one day downloads through DD may be restricted to off market hours.

2) Any time NSE makes some changes at their end, all users are effected and developers are to work on emergency to mitigate the sufferings of users. To reduce the pain to both,

I wish that:

GBC may have another offline mode in addition to the present online mode.

In the offline mode, it will work on the EOD zip files manually downloaded by user and saved in the specified folder and rest of the functioning is normal.

Even if there are any changes at NSE end, the SW will work in offline mode, as long as NSE allows manual downloads, which anyway, it is obliged to do to the trader / investor

base.

Solicit your response on my musings at a later date when the present emergency is over.

Regards.
murthymsr


Hi Hemen,
This idea (offline) seems to be very good. If it can be done then there will be low load on NSE server also and we all will be able to avoid this problem immediately.
Sunil
Reply kkseal
03:50 AM on November 05, 2010 
Hope the Header change suggested by Testman works. (The key i guess is to match exactly the header string in the browser request with that send by GBC I'm sure there are s/w that returns the former There may be property-value pairs other than 'referrer' & 'user-agent' Better to incorporate each of these in the Headers collection of the WebClient object, ensuring no further problem) Let's hope for some good news on Deepavali.

The offline mode suggested by Murthy also merits consideration. Since i can emulate the exact input manually, it should be possible and easier with GBC if the appropriate files are available locally.

I have also posted a discrepancy regarding the Delivery vol of BE series stks.

But all that can wait. The first priority is to get GBC up & running again.
Reply BIJOY
12:14 AM on November 05, 2010 
data down loader of joy is somewhat able to download eod datas..volumediggers.com is the site.But we love getbhavcopy by hemen..we are used to it.Hemen when datadownloader is capable of downloading why getbhavcopy is forbidden...plzz sort out the problem.Bijoy
Reply murthymsr
10:03 PM on November 04, 2010 
Hemen says...
Hi kkseal,

You are on the right track. i already tried that. did not work. :-(

Hi Khokho,

This is a general change by NSE i believe,


Dear Hemen,

I was trying to understand the reasons behind the changes made by NSE resulting in the failure of DD to function normally and the type of modifications required to be done to

Downloaders to bring them back on track.

Looks like that NSE does not want to allow data downloads through SW, but wants to allow through browser only. This may be because the load on their servers is increasing

particularly during market hours.

Code in all downloaders, one by one, will be modified to imitate the browser request and downloaders will start function as before and load on NSE servers will reach the

previous valus and NSE may start looking out for other ways to prevent sw downloads.

It is unfortunate that NSE could have augmented their server capacity instead of restricting sw downloads.

Now to help NSE to reduce the load on their servers during market hours and to serve as a back-up plan, I wish;

1) More than one day downloads through DD may be restricted to off market hours.

2) Any time NSE makes some changes at their end, all users are effected and developers are to work on emergency to mitigate the sufferings of users. To reduce the pain to both,

I wish that:

GBC may have another offline mode in addition to the present online mode.

In the offline mode, it will work on the EOD zip files manually downloaded by user and saved in the specified folder and rest of the functioning is normal.

Even if there are any changes at NSE end, the SW will work in offline mode, as long as NSE allows manual downloads, which anyway, it is obliged to do to the trader / investor

base.

Solicit your response on my musings at a later date when the present emergency is over.

Regards.
murthymsr
Reply Hemen
12:42 PM on November 04, 2010 
User Agent did not work.
Reply VeeEnn
12:08 PM on November 04, 2010 
Hi Hemen,

While you are at it. Could you please incorporate infra and realty indexes from NSE as well in the code.

That will save us time and effort

Thanks
Vee Enn
Reply kkseal
11:40 AM on November 04, 2010 
Have you tried an User-agent string? This seems to be the most common prblm.

Welcome

Categories

Getbhavcopy runs on

Redesign_234x60

Google Search


Recent Forum Posts

by Kunal Shah 12 hours ago
by StockTips 32 hours ago
by Stock Market 36 hours ago
by Manasp 2 days ago

 Subscribe in a reader

 Get email notifications

Recent Blog Entries

12 comments

 Subscribe in a reader

 Get email notifications

Follow me on Twitter

Newest Members