This is from Eric A. Duncan posted elsewhere on cs.org and re-posted here to support Forum FAQs.
As a sidenote, the Roles Provider is not at issue here. The Custom Cookie Authentication does not care what the RolesProvider values are set to. The RolesProvider is only used for the ASP.NET Forms Authentication, which both DNN and CS does use - but in different ways.
You will use the Custom Cookie Authentication (instead of ASP.NET Forms Authentication, which would not work in this case.) Just to be clear, make sure your license is for Custom Cookie Authentication and that you are configuring CommunityServer to use Custom Cookie authentication via the Security Modules download/package.
In DotNetNuke, you will have to create the custom cookie manually in code. This cookie is read by CommunityServer and logs the user in automatically, as well as creating the account if need be.
The easiest place to do this is in the Global.asax file for your DNN Web Application. Tie into the Authenticate_Request() method, which should only fire when the user is attempting to login. At this point you will create a cookie in code matching the paramters you set in the CommunityServer.config file (Cookie name, Username name, Email name).
You can test your settings with the TestAuth.aspx file that ships with the Security Modules. Just place this file in your CS directory and login with it to ensure your CS is setup properly.
Then all you have to do is to create the cookie manually within your DNN on sign-in, and that should be it.