Concept
Full Harvest and Sync
The three kinds of run of the thesis harvest, the night hours, the pauses between requests to the repository and the safety limit for theses missing from the repository.
The thesis harvest reads the repository with three kinds of run. The system starts them by itself in the night hours. You can also start a run by hand on the Catalogue → Thesis harvest screen. Only one run goes on at a time.
Kinds of run
| Kind | What it reads | When |
|---|---|---|
| Full harvest | Every record in the chosen sets | At the first setup and when asked for |
| Sync | The records that are new or changed since the last sync | Every night |
| Removal check | The repository's whole list of records | Every six days |
A full harvest reads every record in the chosen sets from the start and skips the theses that did not change in the repository. Start a full harvest after you change the sets, the record format, the full-text option or the institution name. A sync reads only the records that changed in the repository, so it does not bring the older theses of a newly chosen set.
A sync asks the repository for the records changed since the last successful full harvest or sync. A repository does not always change a thesis's record when its embargo ends, so the sync also reads the theses whose embargo date has passed. One sync checks the embargo of at most 200 theses.
The removal check compares the repository's whole list with the catalogue. A thesis that is no longer on the list is unpublished. A record on the list with a newer date than ours is read again. Some repositories report deleted records only for a while. Thanks to this check, a thesis deleted from the repository is unpublished within a week at the latest.
If the repository's address, the record format or the sets change while a run is going on, the run stops halfway and its problems say the source changed. Start the run again with the new settings.
Night hours
Full harvests and scheduled runs take place in the night hours, when the repository's load is low. By default they are from 01:00 to 06:00, and the Night hours line of the screen shows them. The system administrator changes them in the installation settings.
A full harvest asked for outside the night hours starts when they come. The Sync now and Check for removed theses buttons start their run at once.
Pauses between requests
JournTech sends its requests to the repository one at a time and waits two seconds before each one. When full texts are copied, it waits ten seconds before each file. If the repository's robots.txt asks for a longer pause, JournTech keeps to it up to thirty seconds. If the repository says it is busy, JournTech waits for the time the repository asks for and sends the request again. If the repository names no time, it waits half a minute, then one minute, then two minutes.
The list of records is read page by page, and each page is queued as a job of its own. Every thesis read is saved at once, so the theses read before a harvest stopped are not lost. A run that takes longer than twelve hours counts as failed and is closed.
Safety limit for removals
During maintenance or a migration, a repository may leave part of its records off its list for a while. So that a scheduled check does not empty the catalogue in such a case, there is a limit. If the number of theses missing from the repository's list is greater than both 25 and 5 per cent of the imported theses, the check unpublishes none of them. The run's problems say so, and an alert goes to the system administrators.
If the theses really were removed from the repository, start the check by hand with Check for removed theses. A check started by hand does not apply the limit. The details are in The removal check did not remove the theses.
Failed runs
If the repository cannot be reached, gives an answer that cannot be read, or the run stops halfway, the run ends with the Failed status and an operations alert goes to the system administrators. A problem in a single record does not stop the run.
Nor does a page the repository cannot produce in spite of several tries. The repository usually fails because it cannot write one record on that page. JournTech then gets that set's list of records again from its start and asks for the records it has not harvested yet one at a time. A record the repository still cannot produce is added to the list of problems; the other records are harvested. The next set is read page by page as usual. The solutions are in Thesis harvest looks failed.