Search won't return results that exist in Solr (custom document type)

Search won't return results that exist in Solr (custom document type)

This question is not answered

Hi,

I am integrating a custom application into Telligent Search by following the articles about adding documents to SOL from custom applications here on the Telligent docs.

I have managed to get custom items into the SOLR index and when I query SOLR directly these are returned (e.g: http://localhost:8080/solr1_4/select/?q=test). However when I search on the Community site connected to this SOLR instance I cannot return these custom results. 

Is there something I am missing here that maps SOLR items to those in Telligent. I am guessing the Telligent Search Service does some matching on ContentID / Group / UserId / AppType before it returns results on a Telligent site. Is this correct?

Please explain further how this works and how I can get custom items to be returned.

Here is an example document that I have added to SOLR sucessfully but doesn't show up in search results on the Community site:

<result name="response" numFound="1" start="0" maxScore="2.1027265">
<doc>
<float name="score">2.1027265</float>
<str name="apptype">MyType</str>
<str name="collapse"/>
<str name="content">content of test Solr document</str>
<str name="contentid">999</str>
<str name="createdby">admin</str>
<date name="date">2010-06-28T19:10:33Z</date>
<str name="dummy_id">1</str>
<str name="editedby"/>
<int name="group">5</int>
<date name="indexed_at">2010-06-28T18:10:33.561Z</date>
<str name="rawcontent"/>
<int name="section">10</int>
<str name="title">test doc</str>
<str name="titlesort">test doc</str>
<str name="type">mytype</str>
<str name="unique_id">
   ae383dce-f5e3-4b44-ab12-07ed0f1215c3
</str>
<str name="url">
/mygroup/b/weblog/archive/2010/06/25/test.aspx
</str>
<str name="user">admin</str>
<str name="userdisplayname"/>
<str name="userid">2100</str>
<str name="username">admin</str>
</doc>
</result>

Thanks

Adam

Verified Answer
  • Yes, you need to create a DocumentMapper.  See invalidlogic.com/.../tc5-0-search-adding-your-own-types-part-1 and invalidlogic.com/.../tc5-0-search-adding-your-own-types-part-2 for some further details.

  • Hi Alex,

    Thanks for confirming this (I had stumbled across this article after I posted) but am struggling to get this to work as the documentation is a bit confusing.

    I have writtien a custom Document mapper below:

     public class MyDocumentMapper : DocumentMapper
        {
            public override void Load()
            {
                    Map<MyType>(mytype=>
                {
                   
    mytype.MyTypeId.MapTo(SearchFields.ContentID).Key();
                   
    mytype.MyTypeTitle.MapTo(SearchFields.Title);
                   
    mytype.MyTypeDesc.MapTo(SearchFields.Content);
                   
    mytype.MyTypeDate.MapTo(SearchFields.Date);
                });

                    Rehydrate<MyType>(mytype=> MyType.
    GetTypeById(mytypeid));
            }

            public override int Priority
            {
                get { return 10; }
            }

    MyType class is as follows:

    public class MyType
        {
            public string MyTypeId{ get; set; }
            public string MyTypeTitle{ get; set; }
            public string MyTypeDesc{ get; set; }
            public string MyTypeDate{ get; set; }

            public static
    MyType GetTypeById(object MyTypeId)
            {
                    MyType mytype= new MyType();
                   
    mytype.MyTypeId= "1";
                   
    mytype.MyTypeTitle= "Test Rehydrate Title";
                   
    mytype.MyTypeDesc= "Test Rehydrate Desc";
                   
    mytype.MyTypeDate= "14/06/1982";

                    return mytype;
            }
        }


    I have added the following to the CommunityServer.config Index Task to register my mapper:

    <handler type="BBCommunity.Extensions.MyDocumentMapper, BBCommunity.Extensions" />

    The indexing job seems to be trying to apply my mapper, however I get an error in the event logs about the indexing job erroring:

    Error occurred while running background task SearchIndexJob
    System.NullReferenceException: Object reference not set to an instance of an object.
    at Telligent.Search.Mapping.ContentHandlerTask.Execute()
    at Telligent.Tasks.Task.ExecuteTask()

    I am not sure where to look first as I don't know which part of the code is throwing the "object refernce error" and don't really undestand  how the search index task know which items in SOLR to use this mapping on or how it does this (AppType? maybe). Obvioulsly I just want to apply this to my custom documents.

    Please can you explain a little further and let me know if I am doing anything wrong here so I can progress with this?

    Thanks

    Adam

  • CustomSearchTypeExample.zip

    You had the mapper right but did not create/register a content handler.  

    "..and don't really undestand  how the search index task know which items in SOLR to use this mapping"

    Indexing involved two main components: content handlers and mappers.  At startup the assemblies are scanned to find mappers registered by type. Content handlers are registered via configuration.  The "hooks" for indexing types are in the content handlers. The content handlers are classes that know how to retrieve specific types (ex. blog, users, custom types, etc) that need to be considered for indexing.  Based on the type (or subtype) that was retrieved by the content handler, mappers registered to that type will be called. In your scenerio, you need a MyTypeContentHandler that knows how to retrieve a list of unindexed "MyType" classes.  I have attached an (untested) sample mapper and content handler using your type that you should be able to drop in and go as soon as do the following:

    1. Drop the communityserver_override.config in your web folder.
    2. Provide implementations for the stub methods in MyTypeContentHandler.cs
    3. Add refs to the missing dependencies.

  • Also, the reason you were not getting results in the first post is because you don't have values for "roles" defined on the document.  A document without this defined will never come back in a search since searches are filtered by permissions. ALL searches included the roles you belong to and filter content on that. If you are indexing content that anyone can see, you should still map the everyone role.  An example of this is included in the sample attached to the previous post.

  • Hi Kevin,

    thanks for taking the time to look at this and write some example code for me to work with. I think i misread the section on registering the content handler and accidentally tried to register my contentmappper instead.

    I will try and get this to work as soon as I get in to the office. In the mean time could you please clear up a few things for me:

    1) I followed this example to add custom items into solar, if i am using a contentmapper and contenthandler do i still need to do this? as it seems not if the content handler queries my data store in the GetContentToIndex() method and then indexes it into solr.

    telligent.com/.../integrate-other-applications-with-the-solr-search-service.aspx

    2)  do i need to implement the permissions checks and if so what is this doing exactly?:

    //myType.Group.MapPermissionsTo(_securityService.For(group).GetRolesFor(GroupPermission.DiscoverGroup), SearchFields.Roles);

    3) Will i need to customise the telligent search results widget or searchresultslist control to get it to display my custom document types?

    Thanks

    Adam

  • 1. The telligent.solr library you linked to I believe is for use in an external application - eg pushing search items from a separate e commerce site.  

    2. You would only need to support permissions if your custom items need to be restricted to just certain users.  E.g. If integrating with an e commerce site you most likely wouldn't need to support permissions, however make sure the "everyone" role is granted access as Kevin detailed above.

    3. OOTB your custom items will appear but be listed as 'other'. If you're only adding one type of custom item, you could work around this by editing a resource file or a markup translation.  If you're adding multiple types of search items, then you'll need to modify the main widget.

  • Hi Alex,

    Thanks for confirming these things. A couple more questions on this:

    1. Our application will be embedded in Telligent but at a custom location, e.g: mysite.com/.../itemlisting.aspx

    However we want to index the items in the custom application in the Telligent Search to allow users to find the items and then click on them to go to the custom details page, e.g: ttp://mysite.com/community/myapp/itemdetails.aspx?id=100

    Given this case above is creating a custom content mapper and content handler the correct approach?

    2. I will probably implement my own permissions checking/security module specificly for the application (unless it is possible to add custom permissions to Telligent?)

    Thanks

    Adam

  • A related question. How did you define the integers for Solr? E.g. <int name="section">10</int>. All custom fields I add in the mapper class are saved as strings (<str name="someInteger"...). I need them indexed as an integer in order to do a range search.

    Thanks for any help.

    Best;

    Anders

  • The default field type for unrecognized fields is string.  You have some options:

    ·         Edit schema.config in your solr config directory and add an entry for you in type. Ex.   <field name="mytype" type="tint" indexed="true" stored="true" />

    ·         Name the field with an “_i” or “_ti”  extension.  Ex. “mytype_i”.  Solr has a feature called “dynamic field definitions” in which you can define a pattern to match a fieldname with a type.  “*_i” is the one defined for integers and “*_ti” is one defined for integers that need fast range queries (at the cost of index size).
    Using this options means as well that you will need to include the extension when using this field to search. (ex. mytype_ti:2).

     

     

    Click to show quoted text

  • Its working perfectly. Thanks Kevin!

  • Yes.

All Replies
  • Yes, you need to create a DocumentMapper.  See invalidlogic.com/.../tc5-0-search-adding-your-own-types-part-1 and invalidlogic.com/.../tc5-0-search-adding-your-own-types-part-2 for some further details.

  • Hi Alex,

    Thanks for confirming this (I had stumbled across this article after I posted) but am struggling to get this to work as the documentation is a bit confusing.

    I have writtien a custom Document mapper below:

     public class MyDocumentMapper : DocumentMapper
        {
            public override void Load()
            {
                    Map<MyType>(mytype=>
                {
                   
    mytype.MyTypeId.MapTo(SearchFields.ContentID).Key();
                   
    mytype.MyTypeTitle.MapTo(SearchFields.Title);
                   
    mytype.MyTypeDesc.MapTo(SearchFields.Content);
                   
    mytype.MyTypeDate.MapTo(SearchFields.Date);
                });

                    Rehydrate<MyType>(mytype=> MyType.
    GetTypeById(mytypeid));
            }

            public override int Priority
            {
                get { return 10; }
            }

    MyType class is as follows:

    public class MyType
        {
            public string MyTypeId{ get; set; }
            public string MyTypeTitle{ get; set; }
            public string MyTypeDesc{ get; set; }
            public string MyTypeDate{ get; set; }

            public static
    MyType GetTypeById(object MyTypeId)
            {
                    MyType mytype= new MyType();
                   
    mytype.MyTypeId= "1";
                   
    mytype.MyTypeTitle= "Test Rehydrate Title";
                   
    mytype.MyTypeDesc= "Test Rehydrate Desc";
                   
    mytype.MyTypeDate= "14/06/1982";

                    return mytype;
            }
        }


    I have added the following to the CommunityServer.config Index Task to register my mapper:

    <handler type="BBCommunity.Extensions.MyDocumentMapper, BBCommunity.Extensions" />

    The indexing job seems to be trying to apply my mapper, however I get an error in the event logs about the indexing job erroring:

    Error occurred while running background task SearchIndexJob
    System.NullReferenceException: Object reference not set to an instance of an object.
    at Telligent.Search.Mapping.ContentHandlerTask.Execute()
    at Telligent.Tasks.Task.ExecuteTask()

    I am not sure where to look first as I don't know which part of the code is throwing the "object refernce error" and don't really undestand  how the search index task know which items in SOLR to use this mapping on or how it does this (AppType? maybe). Obvioulsly I just want to apply this to my custom documents.

    Please can you explain a little further and let me know if I am doing anything wrong here so I can progress with this?

    Thanks

    Adam

  • CustomSearchTypeExample.zip

    You had the mapper right but did not create/register a content handler.  

    "..and don't really undestand  how the search index task know which items in SOLR to use this mapping"

    Indexing involved two main components: content handlers and mappers.  At startup the assemblies are scanned to find mappers registered by type. Content handlers are registered via configuration.  The "hooks" for indexing types are in the content handlers. The content handlers are classes that know how to retrieve specific types (ex. blog, users, custom types, etc) that need to be considered for indexing.  Based on the type (or subtype) that was retrieved by the content handler, mappers registered to that type will be called. In your scenerio, you need a MyTypeContentHandler that knows how to retrieve a list of unindexed "MyType" classes.  I have attached an (untested) sample mapper and content handler using your type that you should be able to drop in and go as soon as do the following:

    1. Drop the communityserver_override.config in your web folder.
    2. Provide implementations for the stub methods in MyTypeContentHandler.cs
    3. Add refs to the missing dependencies.

  • Also, the reason you were not getting results in the first post is because you don't have values for "roles" defined on the document.  A document without this defined will never come back in a search since searches are filtered by permissions. ALL searches included the roles you belong to and filter content on that. If you are indexing content that anyone can see, you should still map the everyone role.  An example of this is included in the sample attached to the previous post.

  • Hi Kevin,

    thanks for taking the time to look at this and write some example code for me to work with. I think i misread the section on registering the content handler and accidentally tried to register my contentmappper instead.

    I will try and get this to work as soon as I get in to the office. In the mean time could you please clear up a few things for me:

    1) I followed this example to add custom items into solar, if i am using a contentmapper and contenthandler do i still need to do this? as it seems not if the content handler queries my data store in the GetContentToIndex() method and then indexes it into solr.

    telligent.com/.../integrate-other-applications-with-the-solr-search-service.aspx

    2)  do i need to implement the permissions checks and if so what is this doing exactly?:

    //myType.Group.MapPermissionsTo(_securityService.For(group).GetRolesFor(GroupPermission.DiscoverGroup), SearchFields.Roles);

    3) Will i need to customise the telligent search results widget or searchresultslist control to get it to display my custom document types?

    Thanks

    Adam

  • 1. The telligent.solr library you linked to I believe is for use in an external application - eg pushing search items from a separate e commerce site.  

    2. You would only need to support permissions if your custom items need to be restricted to just certain users.  E.g. If integrating with an e commerce site you most likely wouldn't need to support permissions, however make sure the "everyone" role is granted access as Kevin detailed above.

    3. OOTB your custom items will appear but be listed as 'other'. If you're only adding one type of custom item, you could work around this by editing a resource file or a markup translation.  If you're adding multiple types of search items, then you'll need to modify the main widget.

  • Hi Alex,

    Thanks for confirming these things. A couple more questions on this:

    1. Our application will be embedded in Telligent but at a custom location, e.g: mysite.com/.../itemlisting.aspx

    However we want to index the items in the custom application in the Telligent Search to allow users to find the items and then click on them to go to the custom details page, e.g: ttp://mysite.com/community/myapp/itemdetails.aspx?id=100

    Given this case above is creating a custom content mapper and content handler the correct approach?

    2. I will probably implement my own permissions checking/security module specificly for the application (unless it is possible to add custom permissions to Telligent?)

    Thanks

    Adam

  • A related question. How did you define the integers for Solr? E.g. <int name="section">10</int>. All custom fields I add in the mapper class are saved as strings (<str name="someInteger"...). I need them indexed as an integer in order to do a range search.

    Thanks for any help.

    Best;

    Anders

  • The default field type for unrecognized fields is string.  You have some options:

    ·         Edit schema.config in your solr config directory and add an entry for you in type. Ex.   <field name="mytype" type="tint" indexed="true" stored="true" />

    ·         Name the field with an “_i” or “_ti”  extension.  Ex. “mytype_i”.  Solr has a feature called “dynamic field definitions” in which you can define a pattern to match a fieldname with a type.  “*_i” is the one defined for integers and “*_ti” is one defined for integers that need fast range queries (at the cost of index size).
    Using this options means as well that you will need to include the extension when using this field to search. (ex. mytype_ti:2).

     

     

    Click to show quoted text

  • Its working perfectly. Thanks Kevin!

  • Yes.

  • Hi Kevin,

    I am trying to implement your sample code on a 6.0 site, but it throws an exception on ToMappedDocumentCollection, is this code still valid for 6.0, if not can you point me in the right direction, do you still need to create a ContentHandler & a Documentmapper ?

    Thanks

  • What is the exception?