The best way to differentiate between a built-in field and a custom one is by using the SPField.SourceId Property:
This property retrieves the namespace for built-in fields or, if it is a custom field, the GUID of the list or Web site where it originates from.
In the case of SharePoint REST API, although the SourceId property is not directly available, it can be extracted from the SchemaXml property.
An example below illustrates how to fetch all custom fields from a list:
https://site.sharepoint.com/_api/web/lists/getbytitle('<list title>')/fields?$select=InternalName&$filter=substringof('http://schemas.microsoft.com/sharepoint/v3',SchemaXml) eq false
After retrieving the list of fields, you can gather values of list items by specifying the field names in the $select expression:
https://site.sharepoint.com/_api/web/lists/getbytitle('<list title>')/items?$select=<fieldname1>,<fieldname2>
Update
If you encounter the error message
"Cannot find resource for the request fields"
this indicates that the requested resource (in this case, a library) could not be located. Ensure to provide the library title when using the getbytitle method:
https://site.sharepoint.com/_api/web/lists/getbytitle('<list title>')
^^^^^^^^^^^^^^