The ability to dynamically extend a model to new data and classes is criticalfor multiple organ and tumor segmentation. However, due to privacy regulations,accessing previous data and annotations can be problematic in the medicaldomain. This poses a significant barrier to preserving the high segmentationaccuracy of the old classes when learning from new classes because of thecatastrophic forgetting problem. In this paper, we first empiricallydemonstrate that simply using high-quality pseudo labels can fairly mitigatethis problem in the setting of organ segmentation. Furthermore, we put forwardan innovative architecture designed specifically for continuous organ and tumorsegmentation, which incurs minimal computational overhead. Our proposed designinvolves replacing the conventional output layer with a suite of lightweight,class-specific heads, thereby offering the flexibility to accommodate newlyemerging classes. These heads enable independent predictions for newlyintroduced and previously learned classes, effectively minimizing the impact ofnew classes on old ones during the course of continual learning. We furtherpropose incorporating Contrastive Language-Image Pretraining (CLIP) embeddingsinto the organ-specific heads. These embeddings encapsulate the semanticinformation of each class, informed by extensive image-text co-training. Theproposed method is evaluated on both in-house and public abdominal CT datasetsunder organ and tumor segmentation tasks. Empirical results suggest that theproposed design improves the segmentation performance of a baseline neuralnetwork on newly-introduced and previously-learned classes along the learningtrajectory.