There are three bait parts: feather, hook, and acorn. You can find all three of them by digging up sand tiles at Crab Shore.
When you have all three bait parts, you can make bait, then go fishing.
Just like other rarities, such as shovels, hooks, feathers, and acorns, do not take up space in storage. Collect as many as you want!
To learn more about fishing at Crab Shore, check out the Fishing section.