Concept
DergiPark Request Intervals
At what intervals JournTech sends requests to DergiPark, how it waits while DergiPark is busy and how long-running harvest jobs are split up.
JournTech sends requests to DergiPark (Türkiye's national journal platform) one at a time and with pauses in between. This way, the journal's archive is taken without straining DergiPark. That is why the first harvest of a large archive can take a long time.
DergiPark's rules
The harvest never requests the sign-in and search addresses that DergiPark's robots.txt file forbids. The record list is read from DergiPark's OAI-PMH address in pages of 100 records, and each page is a separate background job.
Waiting between requests
The harvest waits 1 second before each request and 6 seconds before each PDF. The wait is longer for PDFs, because DergiPark limits downloads more strictly than pages.
When DergiPark is busy
If DergiPark answers with 429 (too many requests) or 503, the harvest waits and makes the same request again.
- The waiting time that the response gives (Retry-After) is used.
- If the response gives no time, the wait grows to 30, 60 and 120 seconds.
- The wait never exceeds 5 minutes.
- After such a response, the wait between requests doubles for the rest of that job and can grow up to eight times its normal length.
If there is still no response after the waits, the page of the record list is retried later in the job queue. An article page is read again in the next harvest. A PDF is marked to be tried again later. The details are in DergiPark PDF Copies.
Long-running jobs
Once a job passes 20 minutes, it stops waiting for DergiPark and leaves the remaining PDFs to the next attempt. A run that lasts longer than half a day counts as stuck and is closed as failed.
Only one harvest job runs at a time. When the harvests of two journals are under way at the same time, their pages are taken in turn. Short jobs such as e-mails and virus scans do not wait for the harvest.
Redirects
For a journal without a Turkish interface, DergiPark redirects Turkish addresses to the English page. The harvest follows redirects only within DergiPark and over https, at most three times. The page that answers becomes the original address of the article record.