As we previously announced, we will be moving to an updated version of the E-utilities API for PubMed. In preparation for this launch, a test server is currently available to allow you to test your API calls on the new service and report issues. Thank you for trying out the test server and continuing to submit your feedback!
To address your comments, finalize updates, and to give you more time to prepare for the API update, we are pushing back the release of the new API until later this year.
In the meantime, the test server will remain available and we welcome your continued feedback.
Please refer to our earlier post for more information about the test server and watch our recorded webinar for more details about using the updated API. You can also follow the NCBI Insights Blog and subscribe to the E-Utilities Announcement Mailing List for updates.
Is it possible to reconsider that 10K limit, given more time? Or perhaps to implement some kind of exception for registered users or something?
I hope you will at least be making updates to EDirect to help work around it by using the history server. The epost command as currently implemented in EDirect does not any mechanism to post multiple times to the same webenv, so we will not be able to create a history server collection larger than 10K.
In a real world example of the problem here, my project is using a collection of 500K citations as a benchmarking dataset. I need to upload the PMIDs to the history server, and then search only within those citations. As it stands, this will be impossible via EDirect. We would have to post and search 10K at a time, repeating a set of about 40,000 queries 50 times. That’s a lot of extra queries!
If you are making updates to EDirect, please consider also modifying esearch to return the query expansion or query translation as part of the results. Right now it does not include that info, I assume because it is designed solely for piping the results to some next step. To get the query expansions, we have to use the naked REST interface. EDirect runs a lot more smoothly for us, with the automatic query rate regulating, etc, so we really hope you will expand its capabilities.
EDirect is being updated with a workaround for this limit. Another option is to download PubMed and search it locally using EDirect. See: https://dataguide.nlm.nih.gov/edirect/archive.html
It would be great to have a work around for the 10k limit. If there is any update on that i’d be grateful to hear it. Keep up the good work!
Thank you for your feedback! Please write to info@ncbi.nlm.nih.gov with questions regarding the 10k limit.