The safest way to ensure that you can claim your bags at AMS is to engineer an overnight at AMS, but with the onward flight to DUS less than 24hrs later than the KUL arrival (so that it is treated as a transfer rather than a stopover for fare purposes).
With an overnight in AMS, the agent at KUL will have no problem tagging your bags to AMS rather than DUS. If you can't find a suitable AMS-DUS flight in the relevant booking class, try other German cities. KL fares are normally the same from all German cities and normally allow an open jaw at the origin as long as it is in the same country.