Wednesday, March 21, 2012

Dedupe Priority Logic (within Sort) - feature request?

Is there a way to specify which record is chosen when deduping records using the Sort component? for instance, you might want to choose the last record in for all records with matching sort keys by picking the record with the max(create_date).

By default what record does the sort component pick as the record to be output - first one in finds?

This would be a great feature - a new data flow transform called "Dedupe" that has some powerful deduping features.I think you can achieve what you're talking about by using an Aggregate transform, possibly in conjuction with a Sort.

No comments:

Post a Comment