Yes, your commander can revoke your air assault badge according to AR 600-8-22, para 1-31 c(8):
Any Air Assault Badge may be revoked by the awarding authority (see para 8–26) based upon the determination that an assigned or attached individual has failed to maintain prescribed standards of personal fitness and readiness to accomplish Air Assault missions. Requests for revocation of an Air Assault Badge awarded to individuals no longer assigned or attached to the awarding authority’s command will be forwarded through command channels to Commander, USA HRC, ATTN: AHRC–PDO–PA, 200 Stovall Street, Alexandria, VA 22332–0471. These cases will be limited to those where it has been determined that an awardee was guilty of cowardice, refusal to fly in a tactical exercise or gross negligence in the performance of Air Assault duties.
Please note that HRC is now located at Fort Knox. The regulation does not include the new address.