Thursday, April 8, 2010

Syndicating content from institutional repositories

Institutional repositories powered by software like EPrints and D-Space hold lots of good research content. How do academics potentially interested in this content find it? Obviously, if you're looking for literature on a research topic, say the nature of dispositional properties, you're not going to visit every single IR to do a local search on "dispositional properties". You'll likely do one of three things:
  1. Google "dispositional properties".
  2. Search Google Scholar, Web of Science, or some other generic research index for "dispositional properties".
  3. Search a relevant subject repository for "dispositional properties". In this case PhilPapers would serve you well. Try searching for "dispositional properties" on PP. Not only do you get tons of highly relevant content, but you get a link to a bibliography on dispositional properties maintained by an expert on the subject.
I know based on my logs that a lot of people go for option (3) a lot of the time. PP will generally give you more relevant results than options (1) and (2), and it will often turn up papers that are not even indexed by Scholar or Web of Science & co. That's what it's designed to do.

But my aim here is not to brag about PP's virtues. It's to point out that the content held by IRs can only reach its intended audience if it's properly indexed, and it can only have a maximum impact if it is properly indexed by SRs like PP. But indexing IRs along subject-specific lines is not trivial.

The challenge

How can SRs extract only the content that's relevant to their subject from IRs? Some suggestions:
  1. Ask content producers (academics) to submit their metadata to SRs.
  2. Harvest all content from all IRs, and filter out irrelevant content based on keywords or more advanced document classification techniques.
  3. Crowd-source a list of IRs and OAI sets relevant to your subject.
Of course, all three solutions can be pursued in parallel, but it's worth asking how much content one can expect to get through each of them.

I wouldn't expect much content from (1). Most academics barely have enough energy to submit their papers to IRs; don't ask them to submit them to relevant SRs on top of that. What they could do is submit only to SRs, but this wouldn't be a solution to the problem of syndicating IRs. (Incidentally, we've seen a drastic reduction in philosophy content coming from the few other archives we've been harvesting since PP's launch, so I think the transition to submitting directly to SRs is happening, but there's still a lot of content in IRs we'd like to dig up.)

(2) can't be relied on in isolation. It's a major document classification challenge. However, a lot of papers in IRs have relevant dc:subject attributes. We've tried filtering content from IRs based on the occurrence of the word 'philosophy' in dc:subject attributes. We think that this classification method has a high precision and decent (but far from perfect) recall. I'll post some data shortly.

(3) is also far from perfect, if only because many IRs don't have relevant OAI sets. But a number of them do (for example, the philosophy department has its own set). When that's the case, that's definitely the technique to use.

Our solution

In the end, the solution we've settled for on PP is a combination of all three approaches. We've created an "archive registry" which we've initially populated with all archives listed in OpenDOAR. The registry is not public at the time of writing, but it will soon be. We're currently in the process of filtering out any clearly irrelevant archive (most are).

When we harvest an archive, we first check if it has sets matching certain keywords (our "subject keywords"). If it does, we harvest all and only the contents of these sets. If it doesn't, we harvest everything and filter by matching dc:subject attributes against our subject keywords.

The public registry comes handy when a user notices that their IR-hosted papers aren't making it into PP as they would expect. In this case they can add or edit their IR as appropriate (or they can get their IR manager to do that). Users have the ability to force either set-harvesting or keyword-filtering behaviour. They can also set up a more complex hybrid process combining set- and keyword-filtering. We will bring this system online a.s.a.p and adjust as we see fit based on the feedback we get.

Residual issues

There are other challenges aside from subject-specific harvesting.

One is broken archives. A lot of archives are broken, or have broken content. The most common problem is invalid HTML embedded in the metadata. This can be a serious problem, because a single item can block the harvesting process. If we see that too many archives are stuck like this we'll have to start sending out notices to the admin addresses.

Another issue is that archive metadata are generally incomplete as far as their machine-readable content goes. Typically, either we can't extract publication details or they are missing. That's a major limitation for a site like PP which makes extensive use of such information. I can't think of an easy fix to that. The solution would be good citation extraction tech. I'm not aware of any sufficiently reliable, publicly available algorithm.


113 comments:

  1. [URL=https://www.alrayyanclean.com/index.html]شركة تنظيف بالرياض[/URL] [URL=https://www.alrayyanclean.com/index.html]شركة نظافة بالرياض[/URL] [URL=https://www.alrayyanclean.com/index.html]شركات تنظيف بالرياض[/URL] [URL=https://www.alrayyanclean.com/index.html] شركة تنظيف بالرياض [/URL] [URL=https://youtu.be/rq3ApJgUryQ] شركة تنظيف بالرياض [/URL]

    ReplyDelete
  2. من المعروف ان تنظيف المنازل من بين اهم الخدمات الضرورية في التنظيف والترقية بمنازلنا الى مصاف المنازل العصرية والتي عرفت تنظيفا عصريا من شانه ان يوفر لساكنته ظروفا حياتية راقية ولابد من توفر هيئة تختص في المجال مع تقديم ضمانات متكاملة من شانه ان تجعل الافراد يضمنون نجاح خدمة التنظيف ولا حاجة للمزيد من ضياع الاموال عبر طلبات خدمة تنظيف المنازل التي لا تلبي حاجياتهم ولا تتماشى مع رغباتهم لأن العديد من شركات تنظيف المنازل تسعى فقط الى عرض خدماتها عبر طرق ترويجية فحسب من اجل كسب المال فقط دون مراعاة الوازع الاخلاقي والضمير المهني الذي يحتم على مدراء مثل هاته الشركات ان يسعوا خلف ارضاء العملاء ليس اكثر من هذا عبر اتقان تنظيف المنازل .
    شركة تنظيف بالرياض
    شركة كشف تسربات بالرياض
    شركة تسليك مجارى بالرياض
    شركة عزل خزانات بالرياض
    شركة تنظيف خزانات بالرياض
    شركة تنظيف مجالس بالرياض
    شركة نظافة عامة بالرياض

    ReplyDelete
  3. العاب بنات يحتوي موقعنا على تشكيلة من العاب تلبيس بنات متجددة باستمرار وكل مايتعلق بصنف العاب بنات تلبيس ومكياج و العاب طبخ ومرحبا بكم في العاب تلبيس ...
    العاب بنات - العاب تلبيس بنات ومكياج وطبخ
    al3ab-banat01

    ReplyDelete
  4. تعتبر العاب الفلاش أفضل نوع العاب يحبه الاطفال كثيرا فهو انتشر بشكل سريع وكبير ايضا في جميع مواقع الالعاب هذه الالعاب صراحة رائعة وترفيهية جدا منها المضحكة ومنها المحمسة والكثير
    العاب سيارات 2017 سباق جديدة - jeuxjeux99

    ReplyDelete
  5. In any case, don't bother. Consider about your spending arranging one month from now. Right now is an ideal opportunity to act. cash advance san-diego

    ReplyDelete
  6. Pay down as much as you can in addition to the charges every installment. Paying off the charges will keep the credit sum from expanding; paying more towards standard will bring down the adjust and reduce the expenses for next time. Car Title Loans

    ReplyDelete
  7. Here and there it is known as a loan or a check progress, or a post-dated check advance, or a conceded store check advance. Regardless of how you depict it, a payday advance is a fleeting unsecured high intrigue credit. They by and large range between $100 to $1,000 for times of a couple days to a couple of months. An ordinary payday advance would be $300 for a little while. usacheckcashingstore.com/chula-vista

    ReplyDelete
  8. Other than shrouded charges and unbending terms, borrowers could wind up paying an inflatable installment toward the finish of term. In the event that the credit is not reimbursed inside a month, moneylenders could move it over into another understanding and charge extra expenses and much higher financing costs. A few organizations charge pre-installment punishments that keep borrowers from reimbursing the obligation before time to save money on premium installments. https://www.aaa1autotitleloans.com/chicago

    ReplyDelete
  9. The decide is that moneylenders regard their client's entitlement to security in dealing with their own funds. Ordinarily sites offering a compensation day advances utilize an assortment of safety efforts to keep up the security of your own data. Payday Loans Chicago

    ReplyDelete
  10. My Dad and dad in-law were at both finishes of the range when it came to dealing with their financial records. Father would put in hours, now and again days, finding a two penny mistake in his checkbook enroll. It made him bonkers when his checkbook didn't adjust to the penny with the record proclamation. Cash Advance San-diego

    ReplyDelete
  11. With numerous Internet credit organizations on the web, the Internet likewise gives the whole suite of advance assets and promoters of payday loans. Along these lines, through the Internet, a loan client can do all the aggregate procedure on the web. Cash Advance Chula-vista

    ReplyDelete
  12. Before applying, you should comprehend the money related duties you have.
    Check Cashing

    ReplyDelete
  13. Auto title advance organizations are one more alternatives for here and now cash in view of the value in the individual's vehicle. Car Title Loans

    ReplyDelete
  14. In USA you may find us everywhere, every city and 24/7. We actually love to oblige you the best things with 100% agreements and faster ever.
    check cashing Bridgeport

    ReplyDelete
  15. You have even figured out how to make it reasonable and straightforward to peruse. You have some authentic work ability. Much thankful to you.

    SEO Services Local

    ReplyDelete
  16. Loan specialists are likewise precluded from charging over 36% enthusiasm to military borrowers. While computing the loan cost, extra restoration charges, expenses, benefit charges, or credit protection premiums must be incorporated. Cash Advance San-diego

    ReplyDelete
  17. Though before you may need to sit tight days for an endorsement, no faxing Payday Loans are generally reacted to inside 24 hours. In the event that you are endorsed, the cash will be by and large paid straightforwardly into your ledger that same day, giving you access to the assets in a significantly shorter space of time.
    Payday Loans

    ReplyDelete
  18. At the point when the advance is reimbursed the keys and the title will be offered back to the borrower and in addition the lien being discharged. On the off chance that the borrower defaults on the credit installment, the auto will be reprocessed.

    Auto Title Loans

    ReplyDelete
  19. I would like to say thank you for the amazing details and concepts you are sharing in this.
    Call Girls in Gurgaon
    Call Girls in Delhi
    Call Girls in Aerocity
    Gurgaon Escort Services

    ReplyDelete
  20. I wanted to thank you for this great read!! I definitely enjoyed every little bit of it. I have to bookmarked to check out new stuff on your post.
    Affordable Gurgaon Escorts Roshanikhanna
    http://www.roshanikhanna.com/

    ReplyDelete
  21. It is really hard to say when exactly need to cash your check. Sometimes it may very difficult to cash if it is not official hours. That’s why we feel you much and started 24 hour check cashing services. It is very easy to finding us as we are at your city or nearest places. Just remember us and we will cash your check very short period with a very low cost.24 hour check cashing

    ReplyDelete
  22. very interesting information, I really like it because it can add insight for me more broadly, thank you very much for this extraordinary information
    Cara Mengatasi Iritasi Kulit Akibat Pembalut Saat Haid

    ReplyDelete
  23. This comment has been removed by the author.

    ReplyDelete
  24. یکی بهترین سایت ها برای مطالعه کابینت کلاسیک چوبی سامان چوب است.

    ReplyDelete
  25. واحده من افضل الشركات الرائده في نقل الاثاث داخل الرياض والبلاد المحيطه بها بفضل ماتعتمد عليه من معدات وادوات التي تعتمد علي استخدام التكنولوجيا التي تستخدم في نقل العفش والتي تضمن لجميع عملائنا سلامه العفش الذي ينقل من مكان اللي مكان اخر وعليكم ان تودعو اساليب التلقيديه المستخدمه في نقل الاثاث لانها عمليه لايكن علي الفرد العادي ان يقوم بها فقط عليكم الاتصال بشركه الصفراتشركة الصفرات لنقل الاثاث بالرياض وايضا يوجد لدي شركه الصفرات خدمات اخري مثل خدمات تسليك المجاري وشفط البيارات بجميع انواعها التي تسبب الروائح الكريهه ونعمل علي اختفائها بشكل نهائي ونملك افضل معدات واجهزه تعمل علي معالجه انسداد المجاري التي تسبب ازمه في المكان فقط عليكم الاتصال بشركه الصفرات شركة الصفرات لتسليك المجاري بالرياض كما يوجد ايضا لدي شركه الصفرات خدمه عزل الخزانات من ضمن الخدمات التي تقدمها شركه الصفرات لعملائها كما ان شركه الصفرات تعتبر افضل شركه للخدمات المنزليه وعزل ولحام الخزانات في الرياض وجميع مدن الممكله من خلال ما تملكه من معدات واجهزه متخصصه في المجال وافراد ذوي خبره في المجال فقط عليكم الاتصال بشركه الصفرات شركة الصفرات لعزل الخزانات بالرياض كما يوجد ايضا لدي شركه الصفرات خدمه كشف التسريبات لان تعتبر مشكله تسريبات المياه من المشاكل التي لها اضرار بالغه علي المنازل والتي تخل بالمنازل ويوجد لدي شركه الصفرات مجموعه متميزه من المعدات والاجهزه وايضا العماله ذات الخبره في هذا المجال وبافضل اسعار شامله الخصومات عليكم فقط الاتصال بشركه الصفرات شركة الصفرات لكشف التسربات بالرياض اذا كنت ترغب ف الحصول علي اعلي معدل نضافه للمنازل والفلل والتخلص من الاوساخ والرواسب التي تلازم منزله والتخلص من البكتيريا التي تتسبب في انتشار الامراض عليك فقط الاتصال بشركتنا حيث تجد افضل اسعار وافضل خدمه من خلال شركة الصفرات للتنظيف بالرياض شركه الصفرات من افضل الشركات الموجوده في مجال مكافحه الحشرات بشكل خاص وفي مجالات النظافه بشكل عام ولا يوجد في حياتنا مجال للحشرات للتعايش معانا ويجب ان نتخلص منها بافضل طريقه بحيث الا تظهر مجددا ونتميز نحن بافضل اساليب للتخلص من الحشرات من خلال افضل مبيدات وادويه للتخلص منها عليكم فقط بالاتصال بشركه الصفرات شركة الصفرات لمكافحة الحشرات بالرياض شركه الصفرات تتشرف ان تقدم لعملائها افضل خدمات تنظيف وغسيل وتطهير الخزانات بحيث ان تنظفها بافضل طرق للتخلص من الاوساخ الموجوده بداخلها ونقدم لعملائنا افضل الطرق في تنظيف الخزانات والمحافظه علي تدفق المياه بداخلها وعليكم فقط بالاتصال بشركه الصفرات حيث تجدو افضل الاسعار شركة الصفرات لتنظيف الخزانات بالرياض حيث انه يوجد لدينا جميع ادوات واجهزه التنظيف الحديثه ونتشرف باتصالكم بنا في اي وقت لاننا نعمل خلال 24 ساعه

    ReplyDelete
  26. شركه المثاليه تم تصنيفها علي انها من افضل شركات نقل العفش بالدمام والبلاد المحيطه بها والمملكه باكملها حيث يوجد لدينا افضل عماله متميزه ومتمكنه في هذا المجال وافضل السيارات الناقله للعفش حيث يقومون باشياء لا يقدر علي فعلها الفرد العادي حيث نضمن لعملائنا سلامه العفش خلال عمليه النقل ويتغلف العفش باغلفه تضمن له عدم الخدش او المساس بشكله من جميع النواحي وبافضل اسعار ممكنه لعملائنا عليكم فقط بالاتصال بشركه المثاليه شركة المثالية لنقل العفش بالدمام ويوجد ايضا لدي شركه المثاليه قسم خاص بمكافحه الحشرات حيث لا يوجد مكان للحشرات للتعايش بيننا وعلينا ان نتخلص منها بافضل شكل ممكن بحيث لا تعود مجددا ونتميز نحن بامتلاكنا افضل المبيدات الحشريه التي تقضي عليها تماما وبطرق امنه بحيث لا توذي افراد المسكن وبافضل اسعار ممكنه تجدها لدينا عليكم فقط بالاتصال بنا من خلال شركة المثالية لمكافحة الحشرات بالدمام التنظيف هو عمليه روتينيه تحاتجها المنازل بطريقه دوريه ويجب ان تتم بعنايه فائقه وباجهزه ومعدات حديثه كي تمم بافضل طريقه ممكنه وتحتاج الاماكن الكبيره الي عماله كبيره ذات خبره وهذا ما يتوفر لدينا فالفلل والمنازل والاماكن التجاريه دائما تحتاج الي عماله كبيره ذات خبره فائقه وهذا مايتوفر لدينا وعليكم فقط بالاتصال بنا شركة المثالية للتنظيف بالدمام ولدينا ايضا جميع مستلزمات التنظيف التي يحتاجها جميع الاماكن التي تحتاج الي تنظيف بشكل عام وبافضل اسعار ممكنه يصل الي خصم 40%

    ReplyDelete
  27. tips menggugurkan kandungan seperti apa sih ? Menggunakan tindakan medis seperti Kuret adalah langkah yang terbaik dalam mengakhiri kehamilan . Meskipun banyak beredar di media internet yang menyatakan cara menggugurkan kandungan 100 berhasil dengan bahan alami dengan munggunakan buah pepaya muda maupun nanas muda ataupun dengan bahan tradisional bumbu dapur seperti mengkonsumsi jahe maupun kunyit asam dan adapula yang mempercayai cara menggugurkan kandungan dengan asam jawa , Entah itu hanya sebuah mitos atau opini yang mungkin kebetulan saja . Pahami juga alat kelamin wanita dan fungsinya agar dapat menjaga , merawat dan merapatkan miss v biar menambah keintiman didalam hubungan rumah tangga . Masalah keputihan sangat mengganggu sekali bagi wanita , lantas bagaimana cara menghilangkan keputihan yang terkadang dialami perempuan ?

    ReplyDelete
  28. I Saw Your Website Carefully. Thanks For Create Such Kind of Informative Website. Your All Content Is Relevant To Your Subject. I Say Keep It Continue, Because Your Website And Subject Is Meaningful For The Users. I Have Recommended Your Website With My Friends Also.
    I Am Ritu Sharma. I Deal In Mumbai Escorts Service . I Am Very Hot And Sexy. I Have Five Years’ Experience. That’s why I Understand the Each Personal Needs of My Every Client. My Service Charge Is Low And Service Is Super. You Can Avail My VIP Escorts Service At Your Home Or In Hotel. I Am Comfortable To Provide Escorts Service In Local And Outside Also.
    Visit http://www.escortservicemumbai.net/ Call +91- 9OO4OO9481
    Visit The Links Below And Have A Look At My Various Mumbai Escort Services-
    Mumbai Escorts
    Mumbai Call Girls
    Juhu Escorts
    Andheri Escorts
    Bandra Escorts
    Mumbai Escorts In Vile Parle
    Mumbai Escorts In Colaba

    ReplyDelete
  29. I thank you for the information and articles you provided cara menggugurkan kandungan

    ReplyDelete
  30. This comment has been removed by the author.

    ReplyDelete
  31. Apko humare service all over Gujarat mai milegi apko agar direct contact karna ho toh niche diye gaya whatsapp button ke dwara hume aap contact kar sakte ho. Arjunsharma escortsall over India se ladkiya padhne ati hai jo unka kharcha nhi utha sakti hai unke khud ke pass paise nhi hote hai waise ladkiya unke shok pure karne ke liya ati hai or wo humare genuine kam karti hai hum all over Gujarat mai kam karte hai

    ReplyDelete
  32. thank you for the information provided, we are waiting for the next info at cara menggugurkan kandungan

    ReplyDelete